Distinct Elements is Hard Even for Random Data Streams

We continue the study of approximating the number of distinct elements in the data stream model to within a (1 ± ) factor with constant probability. It was shown by Indyk and Woodruff (FOCS, 2003) that if the stream may consist of arbitrary data arriving to the streaming algorithm in an arbitrary order, then any 1-pass algorithm requires Ω(1/ ) bits of space to perform this task. In an attempt to bypass this lower bound, Chakrabarti et al (STOC, 2008) define a robust streaming model in which the stream may consist of arbitrary data, but it arrives to the algorithm in a random order. However, even in this model the authors were able to show an Ω(1/ ) lower bound. This is because the adversary can still choose the data arbitrarily. We take this a step further and show that even with random data and a random ordering, there is an Ω(1/ ) lower bound. This holds even if each successive stream item is drawn independently and uniformly at random from a subset of items. Our result subsumes all previous ones and shows that this potentially practical assumption of random data does not help. Another relaxation of the problem is to allow multiple passes over the data stream. We also show that even with random data, if in each pass a linear sketch over GF (2) of the data is computed (possibly adaptively), then there is an Ω(1/ ) lower bound. Known approximation algorithms compute such sketches. Previously, nothing better than an Ω(1/ ) bound was known for any class of multi-pass algorithms in any of the above data stream models. Our bound applies to all such models.